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Summary 

For  some  of  today’s  simulations  very  expensive,  heavy, 
and  large  equipment  is  needed.  Examples  are  driving, 
shipping,  and  flight  simulators  with  huge  and  expensive 
visual  and  motion  systems. 

In  order  to  reduce  cost,  immersive  ‘Virtual  Simulation’ 
becomes  very  attractive.  Head  Mounted  Displays 
(HMD)  or  CAVEs  (Computer  Animated  Virtual 
Environments),  Datagloves,  and  cheap  ‘SeatingBucks’ 
are  used  to  generate  a stereoscopic  virtual  environment 
(VE)  for  the  trainee. 

IVS  enhances  training  quality  and  quantity  for 
classroom-teaching  and  Computer  Based  Training 
(CBT).  It  allows  to  visualize  and  animate  teaching- 
material  m a more  natural  stereoscopic  environment. 
Data  of  before  unseen  complexity  can  be  revealed  and 
complex  models  easily  visualized.  For  the  first  time,  the 
trainee  himself  can  interact  with  a Data-Glove  in  the 
environment  and  collect  cockpit  experience  long  before 
his  maiden  flight.  CAVEs  and  Immersive  Projection 
Screens  enable  “group  training”  to  collect  personal  and 
shared  experience  while  further  enhancing  training 
quality. 

With  increasing  maturity  of  VR-gear  IVS  will  allow  to 
generate  new  training  metaphors  for  immersive  flight 
simulation.  This  might  include  the  enhancement  or 
partial  replacement  of  conventional  flight  simulators  by 
IVS. 

Introduction 

High  fidelity  pilot  training  simulators  are  designed  as 
training  tools  for  one  specific  aircraft  type.  They  demand 
authentic  instrumentation  and  system  layout  for  the 
simulated  aircraft  type  including  huge  outside  vision 
systems  and  cumbersome  motion  systems.1  Because  of 
these  reasons,  traditional  simulators  are  very  expensive, 
inflexible,  and  difficult  to  reconfigure.  The  high  cost 
factor  in  buying  and  maintaining  them  causes  air  carriers 
to  purchase  either  just  a single  simulator  for  every 
aircraft  type  they  own  or  to  buy  expensive  training  hours 
from  other  companies.1 

Virtual  Simulation 

To  overcome  some  of  the  problems  in  the  field  of  pilot 
training  the  Air  Force  Institute  of  Technology  developed 


a Virtual  Cockpit  (VC)  for  fighter  pilot  training."  Pilots 
are  immersed  in  a stereoscopic  VE,  wearing  a HMD  and 
a pointing  device  to  interact  with  the  virtual  cockpit 
devices.3  A VC  can  be  easily  reconfigured  by  simply 
switching  the  cockpit  model  database  and  the  attached 
flight  mechanics.4  At  the  Institute  for  Flight  Mechanics 
and  Control,  Darmstadt  University  of  Technology  this 
concept  was  extended  to  be  suitable  for  an  Airbus  A340 
Cockpit-IVS  using  hi-resolution  HMD,  “Seating  Buck”, 
cyberglove,  and  stereoscopic  projection  screens  for  a 
natural  interaction  metaphor.3  As  pilot  outputs  for 
principle  navigation  and  Instrument  Flight  Rule  (IFR) 
testing  a virtual  Primary  Flight  Display  (PFD),  a virtual 
Navigation  Display  (ND),  and  a virtual  civil  Head-Up 
Display  (HUD)  are  available.  In  addition,  a simplified 
outside  visual  is  rendered  to  the  pilot.  These  displays  are 
sufficient  to  run  principle  Instrument  Flight  Rule  (IFR) 
tests  with  the  virtual  cockpit. 

The  problem  of  lacking  force  feedback  in  IVS  was 
significantly  reduced  by  developing  a “Seating  Buck”.6 
Only  side-stick,  pedals,  flap-lever,  and  thrust-lever  are 
physically  available.  All  other  buttons,  dials,  and 
switches  are  simulated  by  simple  plastic  panels.  In  a test 
series  the  concept  and  implementation  proved  to  reduce 
interaction  time  significantly.1,6 

Other  examples  for  operator  training  using  Virtual 
Training  methods  are  Astronaut  training  to  repair  the 
Hubble  Space  Telescope  , submarine  outlook  training  to 
practice  maneuvering  in  a harbor8,  support  pilots 
classroom  education9,  or  caterpillar  training.  Instead  of 
HMDs  very  often  CAVEs10  and  BOOMs11  are  used  to 
avoid  heavy  intrusive  head  gear  and  limited  Field  of 
Views  (FOV). 

Human  Machine  Interface  in  Virtual  Simulation 

Transfer  of  training  from  virtual  into  real  space  still  has 
to  be  proven  for  pilot  training.  For  simple  Cola  can 
sorting  in  a CAVE  transfer  of  training  from  virtual  into 
real  space  was  shown.12,13  Also,  people  trained  in  VR 
have  a better  orientation  in  buildings  than  map  trained 
persons14.  Therefore,  it  can  be  assumed  that  training  in 
virtual  environments  might  be  useful  to  train  trainees  at 
different  requirement  levels. 

Training  quality  limiting  factors  due  to  today’s  hardware 
equipment  such  as  Field  of  View  (FOV)1 5,1 6,1 7,21 , tracker 


1 For  contact  with  authors:  doerr@fmr.tu-darmstadt.de;  schief@fmr.tu-darmstadt.de;  kubbat@fmr.tu-darmstadt.de;  tel.  +49  6151  16 
2890,  fax +49  6151  16  5434 


Paper  presented  at  the  RTO  HFM  Workshop  on  ((What  Is  Essential  for  Virtual  Reality  Systems  to  Meet  Military 
Human  Performance  Goals?  ”,  held  in  The  Hague , The  Netherlands , 13-15  April  2000 , and  published  in  RTO  MP-058. 
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latency18,  presence19,  and  missing  force  feedback1 6 were 
investigated  in  principal.  Only  few  research  exists 
determining  the  limitations  on  training  caused  by  the 
complete  VR-Human  Machine  Interface  (HMI)  design 
and  hardware.20 

VR-HMI  research  has  already  been  conducted 
concerning  force  feedback,  HMD  FOV,  and  HMD 
resolution  for  Cockpit-IVS.1^'21 

A good  general  overview  describing  VR-IIMI  research 
is  presented  in  22 . 

Conventional  Computer-Based  Training 

Computer  Based  Training  and  Procedure  Training  (PT) 
use  PCs  with  a 2D  image,  sound,  a mouse,  and  a 
keyboard  for  interaction.  A trainee  sits  in- front  of  a PC 
screen  and  interacts  by  clicking  with  the  mouse.  CBT  is 
split  into  different  chapters  such  as  radio  navigation, 
flight  planing,  flight  performance,  electronics, 
instrumentation,  and  engines.  Further  enhanced  systems 
allow  partial  simulation  of  functionality.  For  each 
individual  aircraft  type  a different  program  is  available. 
Each  individual  chapter  is  split  into  different  learning 
units: 

• Overview 

• Components  and  Control 

• System  Operation 

• Abnormal  Operation 

• Summary 

• Mastery  Test 

In  different  learning  units  the  trainee  gets  a multimedia 
presentation  of  the  learning  material.  After  the 
introduction,  the  trainee  can  interact  with  the  system  by 
clicking  with  the  mouse  on  interaction  devices.  In  the 
Mastery  Test  multiple  choice  questions  have  to  be 
answered  and  tasks  performed.  The  trainee  can  practice 
all  units  on  his  own  personal  learning  pace.  The  test  can 
also  be  individually  repeated. 

Such  a training  metaphor  helps  to  support  individual 
training.  “Fast  learner”  are  not  frustrated  by  a low  pace 
and  “slow  learner”  are  not  overrun. 


Figure  1:  Flight  management  computer  training 
with  CBT 

The  trainee  does  not  have  any  immersive  experience 
towards  the  real  geometry  and  functionality  of  the 
cockpit.  The  position  of  interaction  devices  in  real  3D 
space  is  unknown  to  him.  Familiarization  in  3D  space 
can  not  be  realized  with  today’s  CBT  systems. 


CBT  with  a VR  Training  Environment 

In  order  to  enhance  CBT,  a 3D  Virtual  Cockpit  model  is 
generated.  All  interaction  devices  such  as  side  stick, 
pedals,  thrust-lever,  knobs,  buttons,  and  dials  are 
modeled  as  3D  geometry.  All  other  parts  and  surfaces 
are  formed  by  simple  textured  geometry.5  This  3D  model 
is  rendered  to  a pilot  wearing  a tracked  high-resolution, 
large  field  of  view  (FOV),  stereoscopic  HMD. 


Figure  2: 1 VS  Cockpit  based  on  modeled  geometry 
and  textures 


For  interaction  the  pilot  wears  a tracked  data-glove 
recognizing  hand  position,  orientation,  and  finger 
bending.  The  trainee  can  virtually  interact  with  all 
cockpit  devices,  dials,  and  buttons.  The  system  response 
on  the  input  can  be  visualized. 
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The  same  image  is  also  rendered  to  a large  stereoscopic 
projection  screen  (Shutter  classes)  enabling  observers  to 
watch  the  trainee  and  his  interaction.  This  allows  later 
discussion  on  the  trainees  performance. 


Figure  3:  Demonstration  room  with  projection  screen 
(three  shutter  signals) 

The  same  concepts  and  structures  known  from  ordinary 
CBT  are  applied.  The  only  difference  is  that  the  trainee 
is  immersed  into  the  scene  allowing  him  to  interact 
naturally  with  his  environment.  Learning  aids  and  eye 
catchers  such  as  symbols,  markers,  and  any  desired 
virtual  information  can  be  visualized  within  the  3D 
virtual  cockpit  as  well.  For  instance,  after  toggling  the 
gear  lever  the  virtual  gear  unit  is  displayed  and  the 
actuator  changes  visualized.  Therefore,  beyond  the  VR 
simulation  of  an  ordinary  cockpit,  virtual  information 
can  be  incorporated  and  used  as  didactical  metaphor  for 
pilot  training. 

Therefore  two  different  Training  methods  are  feasible 
with  this  environment.  In  the  so  called  Class  Room 
Training,  collective  and  cooperative  learning  in  front  of 
a single  projection  screen  enables  trainees  to  work  and 
learn  together  in  the  same  environment. 

As  a second  training  method,  a single  VR-Pilot  training 
environment  was  developed.  Therefore  the  trainee  wears 
a HMD  and  a Data  Glove.  Both  devices  were  tracked 
with  a tracking  system.  So  a naturally  interaction  with 
the  virtual  scene  is  possible.  The  trainee  is  guided  trough 
the  different  lessons  depending  on  his  interaction  with 
the  cockpit  The  trainee  itself  define  which  lesson  he 
wants  to  work  through  . Additionally  it  is  possible  for 
the  trainee  to  fly  with  this  virtual  cockpit  because  of  its 
full  functionality.  Both  Methods  uses  the  same  Training 
lessons  with  minimum  changes  in  interaction 
possibilities.  Example  lectures  were  realized  for  both 
training  methods  and  will  be  described  later  on. 

Classroom  Training 

The  didactical  methods  for  training  vary  depending  on 
the  airlines  and  the  training  facilities.  Training  is  often 
based  on  the  conventional  concept  of  “frontal  teaching”. 
Teachers  give  lectures  with  varying  didactical  materials. 
Dependent  on  the  training  facility  this  can  be  simple 
transparencies,  video-tapes,  sketches,  boards,  and  small 
mockups.  After  each  lecture  the  trainee  has  the 
possibility  to  re-read  the  taught  lecture  from  printed 


material.  At  the  end  of  each  training  chapters  pilots  must 
pass  a written  multiple  choice  test. 


Figure  4:  Stereoscopic  projection  screen 
for  classroom  training 


Hence,  the  understanding  of  complex  aircraft  systems 
strongly  depends  on  the  imagination  of  a trainee  and  the 
teaching  skills  of  a teacher.  In  order  to  enhance  teaching 
quality  stereoscopic  projection  screens  can  be  used  to 
visualize  complex  aircraft  systems  and  technical 
dependencies  in  a natural  way.  A teacher  can  fly  through 
a model,  hide  obstructing  parts,  or  animate  complex 
functionality’s.  The  trainee  himself  becomes  part  of  the 
scene.  In  after  lecture  sessions  the  trainee  can  interact 
with  the  system  and  its  functionality.  In  front  of  an 
immersive  screen  the  trainee  becomes  part  of  the  scene 
and  experiences  the  learn  material  more  naturally. 
Stereoscopic  vision  with  depth  perception  enables  new 
pilots  to  easily  asses  complex  3D  structures,  aircraft 
positions,  etc.  The  “hands-on  experience”  helps  to 
deepen  the  understanding  and  motivate  the  trainee  to 
explore  deeper  into  the  learning  material.  Group 
experience  and  group  training  can  be  enhanced  with  an 
stereoscopic  projection  system.  This  might  accelerate 
memorization  and  pushes  the  later  needed  ability  of 
cooperative  cockpit  work  through  group  experience. 

VR  Pilot  Training 

For  today’s  simulator  training  huge  and  expensive 
simulators  are  needed.  Each  training  hour  costs  up  to 
$5,000  and  existing  training  facilities  are  currently  at  the 
limits  of  their  capacities.  Additional  to  the  later  on 
described  training  lessons,  the  system  is  also  tit  for  use, 
to  train  some  real  flight  tasks.  Therefore,  the  Virtual 
Cockpit  (VC)  based  on  the  above  described  technique 
(HMD  plus  Stereoscopic  Projection  Screen)  can  be  used. 
In  addition  to  the  CBT  approach,  a simplified  outside 
visual  is  added  to  generate  an  immersive  flight 
simulation.  The  viewing  distance  has  to  be  reduced  to 
approximately  20km  in  order  to  ensure  sufficient 
rendering  performance  (15-20Hz).  A virtual  Primary 
Flight  Display  (PFD),  Navigation  Display  (ND),  and  a 
virtual  stereoscopic  Head  Up  Display  (HUD)  are  used  in 
a first  approach.21 
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Figure  5:  Primary  flight  display  and  navigation  display 

These  virtual  displays  show  basic  information  necessary 
to  perform  a controlled  flight  and  allow  basic 
performance  analysis  with  the  system. 

Aircraft  System  Lesson 

Aircraft  system  knowledge  is  one  of  the  main  training 
parts  during  theoretical  pilot  education.  Simple  system 
diagrams  are  used  to  give  the  trainees  an  overview  of  the 
whole  system.  Relations  between  aircraft  subsystems 
and  how  these  systems  work  together  will  be  explained 
in  the  same  way.  This  is  not  a very  intuitive  way  of 
learning.  The  best  way  to  learn  is  to  visualize  them.  The 
visual  channel  is  the  most  significant  wray  to  comprehend 
information. 

The  main  advantage  of  VR-sy stems  is  the  possibility  to 
display  trainees  a 3D  geometry  of  an  object  and  a 
simulation  of  the  real  behavior.  As  an  example  the 
behavior  of  gear,  flaps,  and  rudder  on  an  input  from  the 
pilot  is  shown.  Therefore,  a complete  aircraft  model  is 
shown  to  the  trainee  (Figure  6).  The  model  shows  the 
reaction  of  the  aircraft  and  it  is  possible  for  the  trainee  to 
zoom  in  different  subsystems. 


Figure  6:  Aircraft  outside  view  above  the  pedestal 


Figure  7:  Gear  view 


In  this  example  (Figure  7)  the  trainee  can  observe  the 
kinematics  of  the  gear.  He  can  imagine  the  3D  behavior 
of  the  actuators  during  gear  up  and  down  procedure.  So, 
in  case  of  a system  failure,  he  can  imagine  what  is 
happen  and  where  the  errors  can  be.  The  system 
knowledge  increases  because  of  the  3 dimensional  form 
of  presentation. 

Engine  Lesson 

During  pilot  education  engines  are  visualized  by 
explosion  sketches  or  vertical  cuts  through  an  engine. 
This  creates  a complex  visualization  of  the  parts.  For 
instance,  explaining  the  turbine  turn  rate  at  N 1 and  N2  is 
rather  difficult.  Either  the  graphical  representation  is 
showing  too  much  or  too  view^  detail,  forcing  the 
instructor  to  switch  between  several  images. 


Figure  8:  Conventional  visualization  with  a 
cut  through  an  engine 


Therefore,  a lecture  was  developed  that  allows  to 
dismantle  the  engine  from  a full  blown  representation 
down  to  the  necessary  key  elements  such  as  fan,  turbine 
stators,  turbine  shaft,  and  combustion  chamber.  Trainees 
observe  the  animated  engine  and  can  position  themselves 
on  arbitrary  positions. 
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Figure  9:  3D  engine  inside  view 


Alternatively  the  they  can  follow  a prerecorded  flight 
path  through  the  scene.  During  the  flight  they  can 
arbitrary  change  the  viewing  direction.  They  can  always 
stop  and  move  towards  any  object  to  get  a closer  look. 
To  increase  realism  and  a feeling  for  object  sizes,  a 
complete  aircraft  is  rendered  as  well. 

Stereoscopic  vision  enables  trainees  to  be  immersed  into 
the  environment  generating  a closer  and  better 
impression  of  the  turbine.  It  enables  the  trainees  to 
achieve  a feeling  for  real  part  and  turbine  sizes.  With 
ordinary  paper  sketches  this  is  impossible.  As  an 
enhancement,  observers  can  be  re-scaled  to  small  sizes 
allowing  to  closely  observe  small  turbine  parts  and  their 
functionality. 

Based  on  the  stereoscopic  vision,  instrument  locations 
and  attached  functionality  can  be  memorized  by 
generating  a mental  map  of  the  cockpit 

Force  Feedback/Vision 

It  was  determined  that  lacking  force  feedback  in  pure 
IVS  is  a major  usability  limitation.6  Therefore,  some 
devices  are  physically  available  such  as  sidestick,  pedals, 
and  thrust-lever.  All  others  are  replaced  by  simple  plastic 
panels  to  generate  force  feedback  to  the  pilot  (Seating 
Buck).1  The  Seating  Buck  can  be  easily  reconfigured  to 
simulate  arbitrary  cockpit  configurations. 

With  a Seating  Buck  force  feedback  device  the 
interaction  time  is  reduced  significantly  providing  a 
more  natural  haptical  feedback  to  the  pilot.6 
For  the  success  of  a VR  CBT  enhancement  a large  FOV 
of  more  than  80°  is  needed.21  Above  a 60°  FOV  pilots 
can  assess  all  visible  information  and  geometry  in  the 
cockpit.  Above  a 80°  FOV  also  orientation  and  cross- 


viewing among  two  pilots  simulated  in  the  same  IVS  is 
feasible.21 


Figure  10:  Stereoscopic  projection  screen  rendering 
scene  visible  to  the  pilot 


Figure  11:  Seating  buck  to  simulate  force  feedback 
Usability 

VR  CBT  and  PT  system  can  help  to  reduce  education 
cost  by  reducing  expensive  simulator  hours  for 
familiarization  and  principle  interaction  training. 
Otherwise,  it  can  serve  as  an  extension  to  already  proven 
CBT  training  concepts.  The  VR  technology  and 
projection  screen  technology  is  mature  enough  to  fulfil 
these  tasks.  HMD  with  the  requested  FOV,  force 
feedback  devices,  and  computers  with  sufficient  graphics 
power  exist.  However,  real  verification  of  training 
transfer  has  to  be  further  investigated  in  the  future. 

Flight  Simulation 

All  ordinary  software  simulation  modules  known  from 
conventional  flight  simulation  such  as  physical  input 
devices,  virtual  input  devices,  flight  mechanics,  traffic, 
and  rendering  run  in  a distributed  environment  on 
different  high  end  graphics  work  stations.  As  simulation 
module  an  Airbus  A300  flight  mechanics,  ground 
collision  , weather,  and  sound  modules  are  available.  All 
the  modules  are  taken  from  a conventional  flight 
simulator  available  at  the  Institute  for  Flight  Mechanics 
and  Control.21  It  can  be  also  used  for  comparing  the  VCS 
with  a real  flight  simulation. 
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Figure  12:  Conventional  flight  simulator  mock-up  at 
the  FMRT 


Motion  Base 

In  addition,  to  the  current  approach  of  a fixed  based 
“Seating  Buck”,  the  entire  system  can  be  mounted  to  a 
motion  base.  This  would  increase  the  level  of 
immersiveness  by  aircraft  motion.  To  simulate  a civil 
aircraft  the  performance  of  small  two-seater  (300kg) 
motion  bases  would  be  sufficient.  The  increase  in 
realism  and  immersion  is  untested  and  need  to  be  further 
evaluated. 

Tracker  and  System  Lag/HMD  Limitations 

One  of  the  key  limitations  to  VC  is  today’s  tracker 
latency.  The  entire  VCS  has  currently  a latency  of  about 
100ms.24  From  tests  it  was  deducted  that  150ms  are 
sufficient  for  orientation  tasks  within  the  cockpit/4 
Maximal  lag  for  flying  a VCS  should  be  well  below 
80ms.25 

Another  limitation  is  the  currently  used  HMD  with  a 
FOV  of  maximal  56°.  As  stated  above,  this  HMD 
reduces  the  FOV  in  a way  that  disables  flying  and 
orientation  tasks  within  the  cockpit.  However,  this  is  not 
a general  concept  limitation  because  HMD  vendors 
already  sell  120°  equipment. 

HMD  resolution  is  very  critical  for  the  usability  of  VCS. 
Research  results  indicate  that  with  a hires  HMD  of 
1280x1024  pixels  and  cockpit  displays  (PFD,  ND,  and 
HUD)  of  8 inch  rendered  at  a distance  of  approximately 
85cm  (standard  pilot-display  distance)  offers  sufficient 
resolution.21  Therefore,  resolution  with  modern  HMD  is 
no  more  limitation  to  Cockpit- IVS. 


Usability 

The  used  VR  equipment  is  critical  for  the  success  of 
VCS.  In  principal  most  system  components  such  as  force 
feedback  generation,  HMD  FOV  and  resolution  are 
proven  to  be  usable.  Tracker  lack  has  to  be  reduced 
significantly.  It  is  untested  which  influence  the 
combination  of  VR  equipment  has  on  the  overall 
performance.  The  usability  of  VCS  to  enhance  simulator 
training  is  unproven. 

Even  after  optimization  of  all  VR  equipment  it  seems 
unfeasible  to  completely  replace  flight  simulators  by 
VCS.  In  a first  step  the  understanding  of  the  HMI 
presented  by  intrusive,  heavy,  inconvenient  Virtual 
Reality  gear  has  to  be  further  investigated.  Also,  a 
complete  Virtual  Reality  simulation  theory  is  missing. 
On  the  first  impression  large  projection  screens  have  less 
negative  impact  on  the  HMI. 

Future  Work 

The  Institute  for  Flight  Mechanics  and  Control  will 
further  investigate  the  usability  of  VR-CBT  and  VCS. 
The  research  will  be  focused  on  the  human  machine 
interface  generated  through  virtual  simulations.  The  goal 
is  to  prove  the  usability  (especially  of  CBT)  for  real 
usage  in  today’s  flight  training. 

It  is  assumed  that  the  introduction  of  large  stereoscopic 
projection  screens  into  today’s  pilot  training  will  be  a 
natural  step.  The  equipment  is  already  usable  for 
classroom  training  applications  and  CBT. 

Available  Equipment 

At  the  Institute  for  Flight  Mechanics  and  Control  a 
variety  of  equipment  can  be  used.  A front  projection 
system  (Ampro  Projector)  with  a curved  screen  and  four 
shutter  emitters  is  installed.  For  the  system  seven  shutter 
glasses  (Christal  Eye)  are  available.  For  IVS  a nVision 
Datavisor  10><  (1280x1024),  Kaiser  ViSim500, 

Polhemus  Fastrack,  two  18  sensor  Cyber-Gloves,  and  a 
triple  Pipe  Onyx  (IR)  can  be  used. 
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